{
 "cells": [
  {
   "cell_type": "raw",
   "id": "e9437c8a-d8b7-4bf6-8ff4-54068a5a266c",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 1.5\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0df7646-b1e1-4014-a841-6dae9b3c50d9",
   "metadata": {},
   "source": [
    "# How to stream chat model responses\n",
    "\n",
    "All [chat models](https://v02.api.js.langchain.com/classes/langchain_core_language_models_chat_models.BaseChatModel.html) implement the [Runnable interface](https://v02.api.js.langchain.com/classes/langchain_core_runnables.Runnable.html), which comes with a **default** implementations of standard runnable methods (i.e. `invoke`, `batch`, `stream`, `streamEvents`).\n",
    "\n",
    "The **default** streaming implementation provides an `AsyncGenerator` that yields a single value: the final output from the underlying chat model provider.\n",
    "\n",
    ":::{.callout-tip}\n",
    "\n",
    "The **default** implementation does **not** provide support for token-by-token streaming, but it ensures that the the model can be swapped in for any other model as it supports the same standard interface.\n",
    "\n",
    ":::\n",
    "\n",
    "The ability to stream the output token-by-token depends on whether the provider has implemented proper streaming support.\n",
    "\n",
    "See which [integrations support token-by-token streaming here](/docs/integrations/chat/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a76660e-7691-48b7-a2b4-2ccdff7875c3",
   "metadata": {},
   "source": [
    "## Streaming\n",
    "\n",
    "Below, we use a `---` to help visualize the delimiter between tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84c144d",
   "metadata": {},
   "source": [
    "```{=mdx}\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs />\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "975c4f32-21f6-4a71-9091-f87b56347c33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---\n",
      "Here\n",
      "---\n",
      " is\n",
      "---\n",
      " a\n",
      "---\n",
      " \n",
      "---\n",
      "1\n",
      "---\n",
      " \n",
      "---\n",
      "verse\n",
      "---\n",
      " song\n",
      "---\n",
      " about\n",
      "---\n",
      " gol\n",
      "---\n",
      "dfish\n",
      "---\n",
      " on\n",
      "---\n",
      " the\n",
      "---\n",
      " moon\n",
      "---\n",
      ":\n",
      "---\n",
      "\n",
      "\n",
      "Gol\n",
      "---\n",
      "dfish\n",
      "---\n",
      " on\n",
      "---\n",
      " the\n",
      "---\n",
      " moon\n",
      "---\n",
      ",\n",
      "---\n",
      " swimming\n",
      "---\n",
      " through\n",
      "---\n",
      " the\n",
      "---\n",
      " sk\n",
      "---\n",
      "ies\n",
      "---\n",
      ",\n",
      "---\n",
      "\n",
      "Floating\n",
      "---\n",
      " in\n",
      "---\n",
      " the\n",
      "---\n",
      " darkness\n",
      "---\n",
      ",\n",
      "---\n",
      " beneath\n",
      "---\n",
      " the\n",
      "---\n",
      " lunar\n",
      "---\n",
      " eyes\n",
      "---\n",
      ".\n",
      "---\n",
      "\n",
      "Weight\n",
      "---\n",
      "less\n",
      "---\n",
      " as\n",
      "---\n",
      " they\n",
      "---\n",
      " drift\n",
      "---\n",
      ",\n",
      "---\n",
      " through\n",
      "---\n",
      " the\n",
      "---\n",
      " endless\n",
      "---\n",
      " voi\n",
      "---\n",
      "d,\n",
      "---\n",
      "\n",
      "D\n",
      "---\n",
      "rif\n",
      "---\n",
      "ting\n",
      "---\n",
      ",\n",
      "---\n",
      " swimming\n",
      "---\n",
      ",\n",
      "---\n",
      " exploring\n",
      "---\n",
      ",\n",
      "---\n",
      " this\n",
      "---\n",
      " new\n",
      "---\n",
      " worl\n",
      "---\n",
      "d unexp\n",
      "---\n",
      "lo\n",
      "---\n",
      "ye\n",
      "---\n",
      "d.\n",
      "---\n",
      "\n",
      "---\n",
      "\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "for await (const chunk of await model.stream(\"Write me a 1 verse song about goldfish on the moon\")) {\n",
    "    console.log(`${chunk.content}\n",
    "---`);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61e1309-3b6e-42fb-820a-2e4e3e6bc074",
   "metadata": {},
   "source": [
    "## Stream events\n",
    "\n",
    "Chat models also support the standard [streamEvents()](https://v02.api.js.langchain.com/classes/langchain_core_runnables.Runnable.html#streamEvents) method.\n",
    "\n",
    "This method is useful if you're streaming output from a larger LLM application that contains multiple steps (e.g., a chain composed of a prompt, chat model and parser)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "27bd1dfd-8ae2-49d6-b526-97180c81b5f4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  run_id: \"a84e1294-d281-4757-8f3f-dc4440612949\",\n",
      "  event: \"on_llm_start\",\n",
      "  name: \"ChatAnthropic\",\n",
      "  tags: [],\n",
      "  metadata: {},\n",
      "  data: { input: \"Write me a 1 verse song about goldfish on the moon\" }\n",
      "}\n",
      "{\n",
      "  event: \"on_llm_stream\",\n",
      "  run_id: \"a84e1294-d281-4757-8f3f-dc4440612949\",\n",
      "  tags: [],\n",
      "  metadata: {},\n",
      "  name: \"ChatAnthropic\",\n",
      "  data: {\n",
      "    chunk: AIMessageChunk {\n",
      "      lc_serializable: true,\n",
      "      lc_kwargs: {\n",
      "        content: \"\",\n",
      "        additional_kwargs: {\n",
      "          id: \"msg_01DqDQ9in33ZhmrCzdZaRNMZ\",\n",
      "          type: \"message\",\n",
      "          role: \"assistant\",\n",
      "          model: \"claude-3-haiku-20240307\"\n",
      "        },\n",
      "        tool_calls: [],\n",
      "        invalid_tool_calls: [],\n",
      "        tool_call_chunks: [],\n",
      "        response_metadata: {}\n",
      "      },\n",
      "      lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "      content: \"\",\n",
      "      name: undefined,\n",
      "      additional_kwargs: {\n",
      "        id: \"msg_01DqDQ9in33ZhmrCzdZaRNMZ\",\n",
      "        type: \"message\",\n",
      "        role: \"assistant\",\n",
      "        model: \"claude-3-haiku-20240307\"\n",
      "      },\n",
      "      response_metadata: {},\n",
      "      tool_calls: [],\n",
      "      invalid_tool_calls: [],\n",
      "      tool_call_chunks: []\n",
      "    }\n",
      "  }\n",
      "}\n",
      "{\n",
      "  event: \"on_llm_stream\",\n",
      "  run_id: \"a84e1294-d281-4757-8f3f-dc4440612949\",\n",
      "  tags: [],\n",
      "  metadata: {},\n",
      "  name: \"ChatAnthropic\",\n",
      "  data: {\n",
      "    chunk: AIMessageChunk {\n",
      "      lc_serializable: true,\n",
      "      lc_kwargs: {\n",
      "        content: \"Here\",\n",
      "        additional_kwargs: {},\n",
      "        tool_calls: [],\n",
      "        invalid_tool_calls: [],\n",
      "        tool_call_chunks: [],\n",
      "        response_metadata: {}\n",
      "      },\n",
      "      lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "      content: \"Here\",\n",
      "      name: undefined,\n",
      "      additional_kwargs: {},\n",
      "      response_metadata: {},\n",
      "      tool_calls: [],\n",
      "      invalid_tool_calls: [],\n",
      "      tool_call_chunks: []\n",
      "    }\n",
      "  }\n",
      "}\n",
      "{\n",
      "  event: \"on_llm_stream\",\n",
      "  run_id: \"a84e1294-d281-4757-8f3f-dc4440612949\",\n",
      "  tags: [],\n",
      "  metadata: {},\n",
      "  name: \"ChatAnthropic\",\n",
      "  data: {\n",
      "    chunk: AIMessageChunk {\n",
      "      lc_serializable: true,\n",
      "      lc_kwargs: {\n",
      "        content: \" is\",\n",
      "        additional_kwargs: {},\n",
      "        tool_calls: [],\n",
      "        invalid_tool_calls: [],\n",
      "        tool_call_chunks: [],\n",
      "        response_metadata: {}\n",
      "      },\n",
      "      lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "      content: \" is\",\n",
      "      name: undefined,\n",
      "      additional_kwargs: {},\n",
      "      response_metadata: {},\n",
      "      tool_calls: [],\n",
      "      invalid_tool_calls: [],\n",
      "      tool_call_chunks: []\n",
      "    }\n",
      "  }\n",
      "}\n",
      "...Truncated\n"
     ]
    }
   ],
   "source": [
    "let idx = 0\n",
    "\n",
    "for await (const event of model.streamEvents(\"Write me a 1 verse song about goldfish on the moon\", {\n",
    "    version: \"v1\"\n",
    "})) {\n",
    "    idx += 1\n",
    "    if (idx >= 5) {\n",
    "        console.log(\"...Truncated\")\n",
    "        break\n",
    "    }\n",
    "    console.log(event)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abacb301",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You've now seen a few ways you can stream chat model responses.\n",
    "\n",
    "Next, check out this guide for more on [streaming with other LangChain modules](/docs/how_to/streaming)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Deno",
   "language": "typescript",
   "name": "deno"
  },
  "language_info": {
   "file_extension": ".ts",
   "mimetype": "text/x.typescript",
   "name": "typescript",
   "nb_converter": "script",
   "pygments_lexer": "typescript",
   "version": "5.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
